Lexical Analyzer

A Lexical Analyzer:

Ignores comments
Ignores spaces, tabs, line returns and line feeds
Identifies reserved words (keywords)
Identifies separators such as: (, [, {, etc.
Identifies values: numeric, strings, etc.

The Wintempla lexical analyzer can be used to separate code in tokens. The Cpl::LexicalAnalyzer::Token structure holds information about each token. Wintempla has three lexical analyzers:

C/C++
SQL/PLSQL (Sql::LexicalAnalyzer)
HTML (Web::LexicalAnalyzer)

You can use the Cpl::LexicalAnalizer class as a base class to create your own lexical analyzer using inheritance. You may need to override the virtual methods. As a matter of fact: the SQL lexical analyzer is derived from the Cpl::LexicalAnalizer class.

Token Type

There are several token types. You can inspect the Cpl::LexicalAnalyzer::GetTokenText function to learn about the lexical analyzer. Token types are defined in the Wintempla.h file, they start with LEX_The Wintempla lexical analyzer supports:

Most data types, such as: double, int, etc.
Neural Lab data types, such as: LayerNet, ProbNet, etc,
Most control sentences, such as: if, if-else, for, while, do-while, break, continue, (it does not support switch, but it can be implemented using if-else)

SQL Lexical Analyzer

It is a lexical analyzer specialized in SQL code. You can find the Wintempla SQL lexical analyzer in the SQL namespace. It supports most SQL data types and keywords, such as: SELECT, FROM, DELETE, etc.

Debug

To debug the Wintempla lexical analyzer, you must use the Cpl::LexicalAnalyzer::Debug function. You can use the debug function to find errors and understand how the lexical analyzer operates. To debug the lexical analyzer, you must:

Call Cpl::LexicalAnalyzer::Create
Call Cpl::LexicalAnalyzer::Debug

Running the lexical analyzer

To run the Wintempla lexical analyzer, you must use the Cpl::LexicalAnalyzer::GetNextToken function. The code below shows how to run the lexical analyzer. To run lexical analyzer you must:

Call Cpl::LexicalAnalyzer::Create
Call Cpl::LexicalAnalyzer::GetNextToken

Program.cpp

void Program::Window_Open(Win::Event& e)
{
     Cpl::LexicalAnalyzer::Token lookahead;
     Cpl::LexicalAnalyzer lex;

     if (lex.Create(tbxInput.Text.c_str())==false)
     {
          // Display error
     }
     do
     {
          lex.GetNextToken(lookahead);
          // Do something with lookahead
     }
     while (lookahead.type == LEX_END_OF_FILE);
}

Problem 1

Create a Wintempla dialog application called MyLexical to test the lexical analyzer. Using Wintempla add two textboxes (tbxInput and tbxOutput) with the multi-line property, vertical scrollbars and horizontal scrollbar. Set the event of "Change" in the tbxInput textbox. Set also the "Want return" property in the tbxInput textbox.

MyLexical.cpp

...

void Program::Window_Open(Win::Event& e)
{
}

void MyLexical::tbxInput_Change(Win::Event& e)
{
     Cpl::LexicalAnalyzer lex;
     lex.Create(tbxInput.Text.c_str());
     //
     wstring output = L"Line\tstring\t\tdouble\t\tint/bool \t\ttype\r\n";
     Cpl::LexicalAnalyzer::Token token;
     wstring text;
     while (true)
     {
          lex.GetNextToken(token);
          if (token.type == LEX_END_OF_FILE) break;
          //
          Sys::Format(text, L"%d\t%s\t\t%g\t\t%d\t\t%s\r\n",
               token.line_number, token.string_value, token.double_value, token.int_value, Cpl::LexicalAnalyzer::GetTokenConstant(token.type));
          output += text;
     }
     tbxOutput.Text = output;
}

Inserting tabs in a textbox

By default a textbox does not accept tabs. To learn how to setup a textbox to allow the input of tabs see Wintempla>OOP>Sub-classing.

Por defecto una caja de texto no acepta tabulaciones. Para aprender como configurar una caja de texto para que permita la entrada de tabulaciones vea Wintempla>OOP>Sub-classing.

Problem 2

Create a Wintempla dialog application called HtmlAnalysis to test the HTML lexical analyzer. Using Wintempla add two textboxes (tbxInput and tbxOutput) with the multi-line property, vertical scrollbars and horizontal scrollbar. Set the event of "Change" in the tbxInput textbox. Set also the "Want return" property in the tbxInput textbox.

HtmlAnalysis.cpp

...
void HtmlAnalysis::tbxInput_Change(Win::Event& e)
{
     Web::LexicalAnalyzer lex;
     lex.Create(tbxInput.Text.c_str());
     //
     wstring output = L"Line\tstring\t\tdouble\t\tint/bool \t\ttype\r\n";
     Cpl::LexicalAnalyzer::Token token;
     wstring text;
     while (true)
     {
          lex.GetNextToken(token);
          if (token.type == LEX_END_OF_FILE) break;
          //
          Sys::Format(text, L"%d\t%s\t\t%g\t\t%d\t\t%s\r\n",
               token.line_number, token.string_value, token.double_value, token.int_value, Cpl::LexicalAnalyzer::GetTokenConstant(token.type));
          output += text;
     }
     tbxOutput.Text = output;
}

Problem 3

Create a Wintempla dialog application called SqlAnalysis to test the SQL lexical analyzer. Using Wintempla add two textboxes (tbxInput and tbxOutput) with the multi-line property, vertical scrollbars and horizontal scrollbar. Set the event of "Change" in the tbxInput textbox. Set also the "Want return" property in the tbxInput textbox.

SqlAnalysis.cpp

...
void SqlAnalysis::tbxInput_Change(Win::Event& e)
{
     Sql::LexicalAnalyzer lex;
     lex.Create(tbxInput.Text.c_str());
     //
     wstring output = L"Line\tstring\t\tdouble\t\tint/bool \t\ttype\r\n";
     Cpl::LexicalAnalyzer::Token token;
     wstring text;
     while (true)
     {
          lex.GetNextToken(token);
          if (token.type == LEX_END_OF_FILE) break;
          //
          Sys::Format(text, L"%d\t%s\t\t%g\t\t%d\t\t%s\r\n",
               token.line_number, token.string_value, token.double_value, token.int_value, Cpl::LexicalAnalyzer::GetTokenConstant(token.type));
          output += text;
     }
     tbxOutput.Text = output;
}

Lexical Analyzer